Testing systems and methods

ABSTRACT

A computer implemented method, system and computing device for identifying a test option associated with an application for a user is described. The method comprises selecting a predefined test indicated by a test identifier associated with the requested application, the test having more than one test option associated therewith, generating a hash of the test identifier and a user identifier associated with the user, processing the hash to generate an index, comparing said index with a distribution of numbers divided into multiple ranges, each range being associated with a test option, and selecting a test option associated with the range into which the index falls. The applications may be computer gaming applications.

FIELD OF THE INVENTION

The present invention relates to testing of computer executableapplications.

BACKGROUND

The advent of the internet, and other wide area networks, has led to aproliferation of connected user or client devices, typically runningservices and associated installations from providers of the user'schoice. Such installations may be gaming related. Providers of suchgaming services such as King.com™ are popular, and may have in excess of300 million customers with associated data.

Updates and new offerings or services are common within such anenvironment. These updates, offerings or services are provided to usersby awarding applications provided to the users with different options.The service provider may wish to try out, sample or test these optionsto understand which offerings are more popular or taken up with positivefeedback from the user population before general release. Such a form oftesting comprises so-called “AB Testing” where two groups of customersor users are associated with offerings of option A or option B. A thirdgroup may form a control group. Information about what user has beenassigned to which group is typically held in a database and possibly ata separate server.

When a user logs on to access the service, in order to ascertain whichmodification to deliver, the service provider must also ascertainwhether that user already belongs to a group, and if so, what group theuser belongs to. This can require a call to another server which holdsthis information, and even if not to another server, the call requires aquery to be run over potentially vast amounts of data before the properoption can be returned to the user.

In the context of large customer databases, and associated data suchcalls and queries can be resource depleting and time consuming.

SUMMARY OF THE INVENTION

According to a first aspect, there is provided a computer system fortesting different options for modifying a computer executableapplication delivered to a population of users, the system comprising anapplication server, and a user device associated with a user of thepopulation and configured to communicate with the application server andto access applications from the server, the server having storage forstoring applications for access and at least one test identifieridentifying a test for the application, each test having multiple testoptions for modifying the application, the server having a processorconfigured to, in response to a received request from a user device forthe application: select a test associated with the requestedapplication, generate a hash of a user identifier associated with a userat the user device and the test identifier associated with the selectedtest, process the hash to generate an index value, compare said indexvalue with a distribution of numbers divided into multiple ranges, eachrange being associated with a test option, select a test optionassociated with the range into which the index value falls, and transmitto the user device the application modified by the selected test option.

In an embodiment, the system comprises an event database which holds adata structure comprising at least one user identifier, application datadefining the application, such as a game application identifier, versionnumber and such, and associated event data for that user when executingthe application modified by the selected test option.

In another embodiment, the generation of the hash comprises performingan MD5 hashing operation to provide an integer, for example, 128 bits inlength.

An MD5 hash provides well distributed integers as a result, but otherhashes may also be utilised with a similar effect.

In another embodiment, the processing of the hash to generate an indexvalue comprises performing a modulus operation on said integer.

Advantageously, the modulus operation is a modulo (100) operation toprovide an index value which is an integer falling within saiddistribution of numbers, where the distribution is 0 to 99. Other modulooperations may define other distributions, for example 0 to 9.

In yet another embodiment, the system comprises an analytics server incommunication with the event database and configured to analyse theevent data associated with the test options.

In a further embodiment, stored applications comprise computer games.

In another embodiment relating to the previous embodiment, the testoptions associated with the computer games modify the user interface(UI) of said computer games when delivered to the user device.

Optionally, the test options comprise an AB test. In such an embodiment,each application has two or more test options which represent variantsof the same feature (e.g. UI feature) delivered to the user device. Thetest variants enable the service provider to review user behaviour ineach test group to return an indication of the preferred variant of thefeature.

An analyst wishing to analyse the results of the test once completedwould typically have to look up per user which group or test option theywere previously allocated to. There may be thousands of users whichpartook in a particular test or offer, and running such queries strainscomputing resources, is time consuming and inefficient and may requirespecial access to the application server, therefore providing securityand resource issues which the service provider must contend with.

In another aspect, there is provided a computer device for analysingevent data associated with an application under test, the devicecomprising at least one processor and a database storing: a datastructure storing at least one user identifier associated with a user,application data and event data for the user when executing theapplication modified by one of multiple test options, where in responseto an analysis query comprising a user identifier, the at least oneprocessor is configured to: request a test identifier and associatedapplication event data associated with that user, generate a hash of theuser identifier and the test identifier, process the hash to generate anindex value, compare said index value with a distribution of numbersdivided into multiple ranges to determine within which range the indexvalue falls, select the test option associated with the range into whichthe index value falls, and associate the selected test option with theassociated event data for analysis.

In yet another aspect, there is provided a computer implemented methodfor identifying a test option associated with an application executableat a user device of a user, the method comprising executing a program bya processor which performs the steps of: accessing from a data structurestored in electronic memory a predefined test indicated by a testidentifier associated with the application, the test having more thanone test option associated therewith, generating a hash of a useridentifier associated with the user and the test identifier associatedwith the selected test, processing the hash to generate an index value,comparing said index value with a distribution of numbers divided intomultiple ranges, each range being associated with a test option, andselecting a test option associated with the range into which the indexvalue falls.

The method can be used at an analytics server or an application server:in each case the same test option is independently generated for aparticular user and test.

In an embodiment of the above aspect, an application server modifies theapplication using the selected test option and delivers said modifiedapplication to said user.

In yet another embodiment, event data is generated by monitoringexecution of the modified application at a user device with the selectedtest option.

In another embodiment, an analytics server associates event data fromexecuting the modified application at a particular user device with atest option by generating the test option based on the user identifierand the test identifier.

In another aspect, there is provided a computing device for testingdifferent options for modifying a computer executable applicationdelivered to a population of users from a store for storing applicationsfor access and at least one test identifier identifying a test for theapplication, each test having multiple test options for modifying theapplication, the computing device having a processor configured to, inresponse to a received request from a user device for the application:select a test associated with the requested application, generate a hashof a user identifier associated with a user at the user device and thetest identifier, process the hash to generate an index value, comparesaid index value with a distribution of numbers divided into multipleranges, each range being associated with a test option, select a testoption associated with the range into which the index value falls, andtransmit to the user device the application modified by the selectedtest option.

In yet another aspect, there is provided a computer program on a medium,which when executed by a processor of a computing device, causes saidprocessor to access from a data structure stored in electronic memory apredefined test indicated by a test identifier associated with anapplication executable at a user device of a user, the test having morethan one test option associated therewith, generate a hash of the testidentifier and a user identifier associated with the user, process thehash to generate an index value, compare said index value with adistribution of numbers divided into multiple ranges, each range beingassociated with a test option, and select a test option associated withthe range into which the index value falls.

In another aspect, there is provided a module for determining an indexvalue indicating a test option in response to a provided user identifierand a test identifier by a computing device in communication therewith,the module being configured to generate a hash of the provided useridentifier and test identifier, to process the hash to provide an indexvalue, and to return said index value to said computing device.

In an embodiment of the above module aspect, the generation of the hashcomprises performing an MD5 hashing operation to provide an integer.

In yet another embodiment, the module is further configured to perform amodulo operation on said integer to provide the index value.

The module may be realised in software, hardware or a combination ofboth depending on the particular application.

Other aspects and features are described with reference to the appendedclaims.

BRIEF DESCRIPTION OF DRAWINGS

To understand some embodiments, reference will now be made by way ofexample only to the accompanying drawings, in which:

FIG. 1 shows an example client or user device;

FIG. 2 illustrates a networked computer system;

FIG. 3 shows an example computer system and data structure for holdingevent data;

FIG. 4 illustrates an example table of a number distribution forallocating test options;

FIG. 5 is a diagram illustrating a flowchart of steps for allocating atest option;

FIG. 6 illustrates steps of an embodiment for searching an index; and

FIG. 7 is a flowchart illustrating steps in a method for analyzing whichtest options were provided.

DETAILED DESCRIPTION OF SOME EMBODIMENTS

In the following description, embodiments of a system and method aredescribed which allow a test group to be readily identified withoutcomplex look ups, etc. being needed.

The component wishing to calculate the test group concatenates the userID of the user making the request with the unique name of the test,forming a per-user-per-test identifier. Using this identifier ratherthan just the User ID ensures that the user will be placed intodifferent groups for different tests. The component could be a server,but the same calculation can be done anywhere from the client, theserver, the messaging systems and the analytical systems, and the testgroup can still be determined so long as the test name and the user IDare known.

The component then performs a hashing operation in accordance with e.g.the MDS hashing protocol of the per-user-per-test identifier. Otherhashing schemes may of course be utilized in accordance with therequirements of the service provider. The hashing operationHASH(UserlD:testname1) results in a potentially very large integer (128sbit in the case of MD5) which may be represented by a hexadecimalstring.

The string or integer is submitted to a Modulus operation to reduce itto a decimal number of a more manageable size, generally a numberbetween 0 and 99 using a Modulo(100) operation. The test administratorhas previously divided the range of this number up, and assigned eachsub-range to a test case. For example, 0-9 is assigned to test case A,and 10-99 is assigned to test case B.

Determining the sub-range containing the reduced hash returns the testcase applicable to the requesting user for this test, and enables thecomponent to provide the appropriate test experience to the user. Forexample, if the reduced hash is “8”, then test case A is applicable tothe user and the server will provide the corresponding experience.

A schematic view of a client or user device 100 according to anembodiment is shown in FIG. 1. All of the blocks shown are implementedby suitable circuitry. The blocks may be implemented in hardware and/orsoftware. When implemented in software, this is achieved by executing asuitable program or code sequence by a processor 115. That program orcode sequence can be stored in the memory 120 or downloaded in any knownway. Therefore it is to be understood that other blocks shown in 100 byway of functional blocks could in fact be implemented by suitablyexecuting programs in the processor 115. The user device 100 has acontrol part 110. The control part 110 has one or more processors 115and one or more memories 120. The control part 110 is also shown ashaving a graphics controller 125 and a sound controller 130. It shouldbe appreciated that one or other or both of the graphics controller 125and sound controller 130 may be provided by the one or more processors115.

The graphics controller 125 is configured to provide a video output 135.The sound controller 130 is configured to provide an audio output 140.The controller 110 has an interface 145 allowing the device to be ableto communicate with a network 150 such as the Internet or othercommunication infrastructure.

The video output 135 is provided to a display 155. The audio output 140is provided to an audio device 160 such as a speaker and/or earphone(s).

The device 100 has an input device 165. The input device 165 can takeany suitable format and can be one or more of a keyboard, mouse, touchscreen, joystick or game controller. It should be appreciated that thedisplay 155 may in some embodiments also provide the input device 165 byway of an integrated touch screen for example.

The blocks of the controller 110 are configured to communicate with eachother by an interconnect such as a bus or any other suitableinterconnect and/or by point to point communication.

It should be appreciated that in some embodiments, the controller 110may be implemented by one or more integrated circuits, at least in part.

The user device 100 is shown by way of example only. In alternativeembodiments, one or more of the parts may be omitted. Alternatively oradditionally, some embodiments may comprise one or more other parts.Alternatively or additionally, one or more parts may be combined.

FIG. 2 schematically shows a networked computer system 200. The system200 comprises an application server 220 which delivers applications, forexample games, to user devices 100 over a network 210. The applicationserver comprises a back end infrastructure 240 “BEN” of game player'sdetails, profiles, statistics, and other user associated data andhistory, and an application (for example game) store 245. The BEN 240 isconnected to a database 250 which holds player data to support the backend infrastructure 240. The server 220 has a processor (not shown) whichis responsive to requests from user devices to play games and to deliverappropriate games (in the form of computer executable applications) to auser.

The database 250 also stores user application event data to supportanalytics as will be described later.

The server 220 may communicate via for instance the internet 210 to oneor more client or user devices 100, shown in the Figure by way ofexample as user devices 100 a, 100 b and 100 c, and may further provideconnections to a social network 230 such as Facebook™. User application,game or event data may be associated in some embodiments with theappropriate social network 230 user identifier.

For example, the application store 245 may store and provide fordownload or for online play of games such as the game of applicant“Candy Crush™”.

This particular application has tens of millions of users worldwide, anda provider of such a game may wish to trial out or test new versions ofthe application, or test which offers and features are popular and whichare not so popular before including these in a new release for example.Clearly, no provider wishes to include new features or offers that arenot popular and risk alienating some of the users of that community.

Such offers may also include promotional events such as offering freegame objects such as lives, boosters, new levels and so on.

The testing of such offers, events, promotions or new versions of anapplication may comprise so-called “AB Testing” where two groups ofcustomers or users are associated with different, designated offers “A”,group A, and offer “B”, group B. Another group may typically form acontrol group, (C). “AB” testing can extend to multiple groups A, B, C,D and so on. Usually, an individual will be assigned a group randomly,wherein the testing typically follows strict statistical rulesdetermining population division per grouping. An “offering” is in theform of a test option which modifies the application to present adifferent experience to a user in each group. For example, differentvariants of the same feature can be provided.

In the current systems, when a user logs on to access the service, theservice provider must ascertain whether that user already belongs to agroup, and if so, what group the user belongs to. This can require acall to another server which holds this information, and even if not toanother server, the call requires a query to be run over potentiallyvast amounts of data before the proper option can be returned to theuser.

This can be time consuming, and in particular for applications such as“Candy Crush™” entail the service provider running queries over vastamounts of data stored in the application store 245 or database 250 tofind a user group or test option (or a previously allocated group andoption) and associated events. Typically, such queries strain computingresources and require increased storage space and processing power.

Hence a problem exists in avoiding lookup when determining which optionto deliver to a user partaking in the test.

Similarly, an analyst wishing to analyse the results of the test oncecompleted would typically have to look up per user which group or testoption they were previously allocated to. There may be thousands ofusers which partook in a particular test or offer, and running suchqueries similarly strains computing resources, is inefficient and mayrequire special access to the application server, therefore providingsecurity and resource issues which the service provider must contendwith.

Embodiments will now be described with reference to FIGS. 3 to 7 whichaim to improve upon the above situation.

FIG. 3 illustrates an embodiment of a system of FIG. 2. In addition tothe application server 220, an analytics server 310 having a processor(not shown) and memory or storage 320 is also provided.

According to the underlying principles of the system architecturediscussed herein, there is no centralised store which associates userswith their allocated test options. Instead, the app server 220 andanalytics server 310 each execute a test allocation algorithm (TAA)which generates test options for users on the fly.

However, the algorithm is the same, with the same result. In this way,identifying a test option to deliver to a user and determining whichtest option has been allocated for analysis are decoupled.

The analytics server 310 is typically accessible by statistical analystsor marketers for example, who may be contracted on behalf of the serviceprovider to analyse the results of a test of an application havingvarious offers or options as will be described later.

In this way, the database 250 stores player data 340 to support the backend infrastructure 240. Table 340 illustrates an example data structurewhich may be stored in tabular form, or associated record or array formas well known. In particular, the data structure 340 comprisesassociating a User ID with events in turn associated with gameapplication data. Row 340 a stores a first user identifier (User ID1)and event data associated with game application data relating to anapplication or game that the user plays or has downloaded or registeredinterest in for example. Row 340 b stores a different user identifier(User ID2) with event data associated with the respective gameapplication data, whilst row 340 c stores a third user identifier (UserID3) with an event associated with game application data forapplications that the third user is active in.

For example, if the application is the game Candy Crush™, the event datamay reflect the user accepting an offer such as a new level, or givingfeedback on the offering (such as a score or star rating of a new levelthat was tested), and the application data is the data associated withthat event, e.g. acceptance of the offer, or the feedback rating.

Many other features, events and responses may be stored depending on thestructure of the test and the number of options set by the serviceprovider in a particular test as shall now be described with referenceto FIG. 4.

FIG. 4 shows an example data structure 400 of a test that the serviceprovider wishes to run on one of its applications, such as CandyCrush™.The test has an identifier 405 which in the figure for the sake ofclarity is simply “Testname1”. Of course, the test may be identified invarious different ways such as numerically, or a tag, or simply a numberdepending on how the service provider defines such nomenclature.

Testname1 405 has, in this example four options 410 associated with it.For example, the service provider may wish to try out various offers ofgift items within a game and deduce which offer/item is the mostpopular. For example option1 450 may be the provision of extra lives toa user. Option2 460 may comprise the ability for the user to “gift”these lives to a friend together with an invite to that friend to playthe game. Option3 470 may comprise a different “gift” offering in theform of the number of lives that can be gifted, whilst Option4 480 maycomprise the ability to offer the gift to more than one friend.

Hence in this test 400, a number of users are needed to participate inoption1 which is designated the Control group, whereas options 2, 3 and4 are designated groups A, B, and C 440.

In this form of A/B testing the provider wishes to allocate the optionsrandomly to users according to the probability distribution 420, i.e.10% of the sample or population should be allocated in the control group(option 1), 20% of the sample or population should be allocated in groupA (option 2), 20% of the sample or population should be allocated ingroup B (option 3), and 50% of the population should be allocated ingroup C (option 4).

As can be seen in FIG. 4, each option 410 is further provided with anassociated number range 430. In this embodiment the range spans 0 to 99,and each option 410 is apportioned with a number range in proportion tothe assigned probability 420. Hence, option 1 is associated with therange 0-9 in accordance with the 10% probability associated with theoption, likewise option 2 is associated with the number range 10-29 inaccordance with the 20% probability also associated with the option,option 3 is associated with the range 30-49 in accordance with the 20%probability also associated with the option, and option 4 is associatedwith the number range 50-99 in accordance with the 50% probability alsoassociated therewith.

The use of this data structure will now be described with reference toFIG. 5.

FIG. 5 illustrates steps of an embodiment, performed by the TAA atapplication server 220 to allocate a test option 410 of a test 405(“Testname1”) to a user device 100.

At step 510 an electronic data message comprising a request for anapplication or associated offer is received by application server 220and the user ID, which may be a login identity of the user or a deviceidentifier, or some combinations of both, is stored 240, 250.Application server 220 subsequently, at step 520 selects a test 400associated with the application from database 250, the test having anidentifier, for example “Testname1” or some other suitable identifier.

At step 530 the server 220 performs a hashing operation of the user IDwith the test identifier to provide a unique string of data which may befor example represented in hexadecimal form. Such hashing operations arewell known. For example, in an embodiment the well-known “MD5” hashingoperation may be performed.

At step 540 the hashed data is processed to provide an index value ornumber. Hence the combination of the userID1 with the Testname1 issuitably reduced to a number or index value in a predefined range 430.

An example embodiment of the processing 600 of the hash is shown in FIG.6, which will be described later.

The server 220 then proceeds to step 550 wherein the index value ornumber produced is compared with the data for that test 400 storingnumber ranges 430 associated with test options 410 and respectiveprobabilities' 420.

The server 220, at step 560, selects a test option, and hence associatedtest group 440 based on the comparison. For example, if the hashingprocedure and processing results in the number or index value “37”,then, with reference to FIG. 4, option 3 is selected as 37 falls withinthe range 30-49 and corresponds to group B for that test. In otherembodiments, for different desired probability distributions, the rangemay be varied.

That test option, including the encoded offer or gifts for example isthen used to modify the application which is transmitted to the userdevice 100. Such modifications can be pre-stored in the database 250,for example whole levels of a game application in a suitable format.

Subsequently, but not shown in FIG. 5, any interaction by the user withthe test option which has been incorporated in their version of theapplication which generates an event and associated game data is storedas shown and described with reference to FIG. 3. This data does notnecessarily have to be associated with the particular test option atthis stage. It is enough that it is associated with the user ID becausethe test option which was allocated can be determined by implementingthe TAA at the analytics server 310.

FIG. 6 is an example embodiment of a method or algorithm for theprocessing of the hashed data (step 540) to provide an index value ornumber.

At step 610 the server 220 performs a hashing operation HASH((UserlD:test identifier) in accordance with the MD5 hashing protocol ofthe user ID making the request. Other hashing schemes may of course beutilized in accordance with the requirements of the service provider.The hashing operation is preferably chosen to convert the user ID: testidentifier into a well distributed integer, for example, an MD5 hash.The well distributed integer which is produced is processed to reduce itdown to a test range in any suitable way. In the present case it is doneby submitting the string or integer at Step 620 to a Modulus or Modulo(100) operation to provide a decimal number or index value in this casebetween 0-99. However other ways are possible.

In one embodiment the last two digits of the result may be selected andthe rest discarded to return an index value within the range of thedesired population.

At step 630 the index value is returned.

Advantages of the above methods lie in efficiently allocating a user toa test group in a manner that is computationally efficient and two-way,avoiding absolute look-up of users and their associated groups,especially once a test option has been assigned, since the test optionmay be returned by simply running the same hashing and indexingprocedure at the analysis server rather than searching a database forthe User ID and associated groups.

The two-way nature of the scheme will now become apparent with referenceto FIG. 7 which shows steps of an embodiment 700 carried out by way ofexample by the TAA at the analytics server 310. This may occur when ananalyst makes a request to analyze a previous or ongoing test to produceresults and a conclusion for the service provider 220. To do this, theanalyst must know which test option and hence group a user whomparticipated in the test was allocated to.

The analytics server 310 provides a test identifier associated with anapplication (e.g. “testname1” as shown and described with reference toFIG. 4 previously) at step 710, and subsequently at step 720 requestsUser ID and associated event data, which is provided via the datastructure 340 of FIG. 3.

At step 730 the analytics server performs the hashing operationHASH(UserlD:test identifier) as previously described, such as a MD5hashing operation. The result is subsequently processed, such asdescribed with reference to FIG. 6, to provide an index value at step740.

The analytics server then compares, at step 750 the computed index valuewith the number ranges associated with the test 400 and determines thetest option that the user was provided with based on the range withinwhich the index value or number falls.

Therefore, the group in that test to which the user belongs, beingassociated with the option is determined based on the index. Thedetermined test option and associated event data is then provided to theanalyst for subsequent analysis.

In an embodiment the data structure 400 describing the test options andprobability ranges and groups therefore may be provided in the memory320 or store of the analytics server 310, therefore improving processingand requiring database queries for only a user ID and event dataassociated with the application under test.

In another embodiment, there is provided a module for determining anindex value indicating a test option in response to a provided useridentifier and a test identifier by a computing device in communicationtherewith, the module being configured to generate a hash of theprovided user identifier and test identifier, to process the hash toprovide an index, and to return said index to said computing device.

In this embodiment, the generation of the hash may comprise performingan MD5 hashing operation to provide an integer.

In another embodiment, the module may be configured to perform a modulooperation on the integer to return or provide the index.

In a variation of the above embodiment, the modulo operation is aMOD(100) operation, with the last two bytes of the integer being used todetermine an index value lying between 0-99.

The module may be provided with a network interface, which may be wired,wireless, cellular or optical to enable communication with a computingdevice itself configured to communicate with the module via appropriateor compatible API and network interfaces.

In some embodiments, the computing device may provide either, or both ofthe test identifier and user identifier to the module.

Hence, a system and algorithms or schemes are provided which allocate auser to a test option in a manner that enables the application server220 to be decoupled from the analysis server 310 and hence avoidsexcessive queries and look-ups.

In addition, the anonymity of the users is preserved, whilst enablingqueries which are analysed algorithmically in an efficient manner toreduce back end load and query time.

Other embodiments may comprise different modulo operations depending onthe population size for example.

Since the embodiments above decouple the analysis process from the userdevice and application server, the data can be modified or updated asrelevant events occur and requests from the analyst input withoutcausing resource depletion.

A person skilled in the art will realise that the different approachesto implementing the systems, devices schemes and methods disclosedherein are not exhaustive, and what is described herein are certainpreferred embodiments. It is possible to implement the way in a numberof variations without departing from the spirit or scope of theinvention.

1. A computer system for testing different options for modifying acomputer executable application delivered to a population of users, thesystem comprising an application server, and a user device associatedwith a user of the population and configured to communicate with anapplication server and to access applications from the applicationserver, the server having storage for storing applications for accessand at least one test identifier identifying a test for the application,each test having multiple test options for modifying the application,the server having a processor configured to, in response to a receivedrequest from a user device for the application: select a test associatedwith the requested application, generate a hash of a user identifierassociated with a user at the user device and the test identifierassociated with the selected test, process the hash to generate an indexvalue, compare said index value with a distribution of numbers dividedinto multiple ranges, each range being associated with a test option,select a test option associated with the range into which the indexvalue falls, and transmit to the user device the application modified bythe selected test option.
 2. A computer system according to claim 1,comprising an event database which holds a data structure comprising atleast one user identifier, application data defining the application andassociated event data for that user when executing the applicationmodified by the selected test option.
 3. A computer system according toclaim 1, wherein the generation of the hash comprises performing an MD5hashing operation to provide an integer.
 4. A computer system accordingto claim 3, wherein the processing of the hash to generate an indexvalue comprises performing a modulus operation on said integer.
 5. Acomputer system according to claim 4, wherein the modulus operation is amodulus(100) operation to provide an index value falling within saiddistribution of numbers.
 6. A computer system according to claim 1,further comprising an analytics server in communication with the eventdatabase and configured to analyse the event data associated with thetest options.
 7. A computer system according to claim 1, wherein thestored applications comprise computer games.
 8. A computer systemaccording to claim 7, wherein test options associated with the computergames modify the user interface of said computer games.
 9. A computersystem according to claim 1, wherein the test options comprise an ABtest.
 10. A computer system according to claim 9, wherein the AB testreturns an indication of the preferred test option.
 11. A computerdevice for analysing event data associated with an application undertest, the device comprising at least one processor and a databasestoring: a data structure storing at least one user identifierassociated with a user, application data and event data for the userwhen executing the application modified by one of multiple test options,where in response to an analysis query comprising a user identifier, theat least one processor is configured to: request a test identifier andassociated application event data associated with that user, generate ahash of the user identifier and the test identifier, process the hash togenerate an index value, compare said index value with a distribution ofnumbers divided into multiple ranges to determine within which range theindex value falls, select the test option associated with the range intowhich the index value falls, and associate the selected test option withthe associated event data for analysis.
 12. A computer implementedmethod for identifying a test option associated with an applicationexecutable at a user device of a user, the method comprising executing aprogram by a computer processor which performs the steps of: accessingfrom a data structure stored in electronic memory a predefined testindicated by a test identifier associated with the application, the testhaving more than one test option associated therewith, generating a hashof a user identifier associated with the user and the test identifierassociated with the selected test, processing the hash to generate anindex value, comparing said index value with a distribution of numbersdivided into multiple ranges, each range being associated with a testoption, and selecting a test option associated with the range into whichthe index value falls.
 13. A computer implemented method according toclaim 12, further comprising modifying the application using theselected test option and delivering said modified application to saiduser.
 14. A computer implemented method according to claim 13, in whichevent data generated from executing the modified application at a userdevice is associated with the selected test option.
 15. A computerimplemented method according to claim 14, which when executed by ananalytics server requests said associated event data with the selectedtest option for analysis.
 16. A computer implemented method according toclaim 12, wherein the generation of the hash comprises performing an MD5hashing operation to provide an integer.
 17. A computer implementedmethod according to claim 12, wherein the processing of the hash togenerate an index value comprises performing a modulus operation on saidinteger.
 18. A computer implemented method according to claim 17,wherein the modulus operation is a modulus(100) operation to provide anindex value falling within said distribution of numbers.
 19. A computingdevice for testing different options for modifying a computer executableapplication delivered to a population of users from a store for storingapplications for access and at least one test identifier identifying atest for the application, each test having multiple test options formodifying the application, the computing device having a processorconfigured to, in response to a received request from a user device forthe application: select a test associated with the requestedapplication, generate a hash of a user identifier associated with a userat the user device and the test identifier, process the hash to generatean index value, compare said index value with a distribution of numbersdivided into multiple ranges, each range being associated with a testoption, select a test option associated with the range into which theindex value falls, and transmit to the user device the applicationmodified by the selected test option to the user device.
 20. A computerprogram on a non-transitory computer-readable medium, which whenexecuted by a processor of a computing device, causes said processor toaccess from a data structure stored in electronic memory a predefinedtest indicated by a test identifier associated with an applicationexecutable at a user device of a user, the test having more than onetest option associated therewith, generate a hash of the test identifierand a user identifier associated with the user, process the hash togenerate an index value, compare said index value with a distribution ofnumbers divided into multiple ranges, each range being associated with atest option, and select a test option associated with the range intowhich the index value falls.
 21. A module for determining an index valueindicating a test option in response to a user identifier and a testidentifier provided by a computing device in communication therewith,the module being configured to generate a hash of the provided useridentifier and test identifier, to process the hash to provide an indexvalue, and to return said index value to said computing device.
 22. Amodule according to claim 21, wherein the generation of the hashcomprises performing an MD5 hashing operation to provide an integer. 23.A module according to claim 22, wherein the module is further configuredto perform a modulo operation on said integer to provide the indexvalue.
 24. A computing device configured to communicate with the moduleof claim 21.